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SECTION I 



CONTEXT, FOCUS AND OBJECTIVES 
OF THIS PAPER 

We have all heard some horror stories about evaluation studies 
that never got off the ground; about the choice of evaluation questions 
that made no sense in terms of program needs-, about evaluation efforts 
that were launched with lot of fanfare but were never completed, or 
were completed too late for use in program improvement; or about evalu- 
ation reports that once completed were put on the shelves where they 
collefcted dust over the years. 

Vihat Is This Paper All About? 

This paper seeks to provide happy endings to some of the horror 
stories just recounted. This paper will deal with evaluation planning 
(so that appropriate and timely initiatives can be undertaken for 
evaluating literacy campaigns, programs and projects, and the right 
evaluation questions can be chosen); evaluation management (so that 
evaluation studies once undertaken can be completed effectively, 
efficiently and on time for use in program improvement); and utilization 
of evaluation results (so that knowledge gained and lessons learned 
from evaluation studies can be put to work in decision making to improve 
program performance). 

It should be stated that the methodology /and technology -^f evalua- 
tion are not of substantive interest to us in fKjs paper. Topics such 
as models of evaluation, evaluation methodology, indicator writing and 



instrumentation have been discussed elsewhere and the reader*s attention 
is invited to those sources J In the body of the present paper, these 
topics wJl be discussed only insofar as they relate to the processes of 
evaluation planning, evaluation management, and utilization of evaluation 
results and findings. 

The Focus and Basic Objectives of the Paper 

« The focus of this paper is on internal evaluation ~ built-in 
evaluation that is fully integrated with the overall processes of pro- 
gram planning and program implementation. The assumption is that 
evaluation will be conducted continuously, by agents ^\iternal to the 
program system, to generate feedback that can be put to use in improving 
the program. Such evaluation will increase organizational intelligence 
and the capacity of the system to perform better. (Most of the ideas 
discussed below can, of course, be applied to external evaluation as 
well.) 

There are three general objectives of this paper: 

Instructional objective . This is to demonstrate to the reader tne 
systematic process of doing evaluation planning; to present the set of 
management problems involved in the implementation of evaluation studies; 
and to propose actions that must be followed through if feedback 
generated by evaluation has to be put to work in decision making. 



Bhola, H.S. Evaluating Fi ^ncuional Literacy . Amersham, Bucks 
(U.K.): Hulton Educational Publications, 1979, 154 pages. Also Bhola, 
H.S. Evaluating Development Tra:nino Programs . Bonn, FRG: German 
Foundation for International Deve^npmont, 1982, 291 pages. 
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Progranmatic objective . This is to suggest to the participants of 

0 

the Workshop what in-ight be some of the evaluation questions that should 
be part of their current evaluation plans, within the particular con- 
texts of their programs of literacy or nonformal education in their 
countries; and what are the specific management and utilization 
Strategies that they must use for efficient and effective evaluation. 

Utilitarian objective . This is a practical, rather mundane objec- 
tive: to present in one, somewhat brief, document most of the important 
ideas available to the profession on the topics of evaluation planning, 
evaluation management and utilization of evaluation results. This is 
done for the benefit of the group gathered at Kany, Botswana for the 
short ten-day period so that they can "hit the ground running"; and 
accomplish the practical tasks of developing evaluation, management and * 
utilization plans for the programs they represent. 

Who Is This Paper Addressed To? 

This paper does seek to have an audience larger than those partici- 
pating in this Workshop. 

More importantly, this paper is not addressed only to the evaluation 
types the few^ people carrying evaluation related designations such as 
di)*ector of evaluation, evaluation officer, evaluation specialist, 
evaluation assistant, statistician, data analyst, field investigator or 
survey assistant, etc.; and who perphaps work full-time on evaluation 
tasks. 

We address this paper to aJ2 those who are part of the evaluation 
subsystem wit hi^ the overall program system. This evaluation subsystem. 



8 



we assert, will include f> whole ariny of program people who work only part 
of the time on evaluation tasks but who perform the most important and 
indispensable support roles and interface rol es. The program people in 
support roles 'provide evaluators the needed administrative support; and 
collect, systematize and contribute the routinely-generated program 
information to the evaluation effort. Again, program people in inter- 
face roles link the evaluation subsystem with other subsystems within 
the overall program system. These support and interface roles go from 
literacy teachers and supervisors in the field, through program 
specialists and program planners, all the way up to policy makers who 
make important political and budgetary decisions. 

The Right Conception of an Evaluation Subsystem 

Too many evaluation efforts are doomed to failure because of a 
defective conception of the evaluation subsystem in relation to the 
total program system. Unfortunately, some evaluators have too exclusive 
a conception of their roles. They are the only evaluators and everybody 
else is there to answer their questions, to do their bidding, feel awed 
and act respectfully. They fail to realize that while there are a few 
full-time evaluation officers (FTEO's) within the system, there are 
many many more full-time program officers (FTPO*s) who are playing 
significant, though, part-time evaluation roles . In terms of man-hours 
spent their input into evaluation will indeed be many times more than 
the man-hour input of whole-tim.e evaluators. 

If the FTEO's do not understand the important role of FTPO's in 
evaluation tasks, and then do not give them due credit for their contri- 
butions, evaluators get cast into parasitical roles, they are denied 



access to evaluative information and program data, and the overall 
evaluation effort collapses. 

We strongly suggest that the evaluation subsystem within the larger 

program system be conceptualized as shown in the figure Delow: 

o 
o o 
. • oo 

••ooooooo 
•••ooooooo 



o 

o o 



Figure I.l: The organization of an overall program 
system showing relationships between the evaluation 
subsystem and the program system 

(jjj^ FTEO's — Full-time evaluation officers 

FTPO's ~ Full-time program officers 

FTPO's in support and interface roles in regard to the 
^ evaluation effort. 

To our particular conception of the evaluation subsystem, we must 
add an additional statement. This is about our special orientation to 



10 



the processes of planning, management and utilization of evaluation 
results. This orientation can be best described as qualitative. We use 
intermediate technology and qualitative models and methodologies of 
planning, management and kno\^ledge production and utilization. The 
approach used, while qualitative^ is both systematic (presenting step by 
step procedures) and systemic (based on systems theory and, thus, on 
system thinking). We suggest that readers "think with" the various 
schematas and tabulations that follow, as they engage in the complexities 
(and the dialectic) of planning and decision making in the real world. 

Literacy Campaigns, Literacy Programs and Literacy 
Projects: Three Faces of Adult Literacy Promotion 

A campaign is any large-scale s^'ehes of activities, intensely 
focussed on a set of objectives to be achieved within some pre-determined 
period of time. Literacy campaigns are typically born of heightened 
political commitment, seek full engagement of the masses and have recon- 
structionist objectives changing old societies into new ones. 

A program is any proposed or pre-arranged course of action and plan 
of activities* A literacy program (like a literacy campaign) may be 
large-scale, focussed, urgent and time-bound. Perhaps the only difference 
between a literacy campaign and a literacy program is in terms of their 
orientation to the planning and management of the ctynamics of change 
involved* A literacy program is conceptualized primarily as profession- 
ally organized development action rather than as political action; it 
follows a different style for the involvement of the masses with the 
program; and can be said to have reformist objectives the revolution 
is already a part of history! 
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A project is a well-defined problem that must be solved or a task 
or piece of work that must be accomplished. A literacy project (unlike 
a literacy campaign or a literacy program) is relatively speaking not 
large-scale; though it will be often focussed and can be both urgent 
and time-bound. The "means and ends calculus" of a project is akin to a 
literacy program rather than to a literacy campaign. Indeed, a literacy 
program will often be composed of many literacy projects. 

In sum, while literacy campaigjrs, literacy programs, and literacy 
projects do differ from each other in regard to the political versus 
> professional definition of social action, the heat of commitment and the 
style of popular mobilization, these three approaches to the eradication 
of adult illiteracy are all alike in terms of their underlying 
strategic design . That is, literacy campaigns, literacy programs and 
literacy projects all have to draw upon the same set of theories, models, 
principles and strategies for their planning, management, implementation 
and evaluation. 

What we are saying here is that the discussion that follows on the 
topic§\9f eveiluation planning, evaluation management and. utilization of 
evaluation results ^pplies equally well to literacy campaigns, literacy 
programs and literacy prpjects. 

. Scope of Application of the Paper: 
L arger than Literacy, Farther than "Kang" 

We have used literacy campaigns, literacy programs and literacy 

projects as exefnplars in 6ur discussion of the evaluation planning, 

evaluation management and utilization of evaluation information. The 

discussion, however, applies also to most out-of school settings (and 
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even to in-school settings if the study of the broad impact of education 
is part of the concern). Thus, the paper covers all adult basic educa- 
tion programs in out-of school settings and can be easily adapted to 
most nonformal education programs conducted within various developmental 
sectors such as agricultural extension, family planning, cooperatives 
education, correspondence education, education and work, etc. 

Again, while participants to the seminar in Kang have come only 
from Botswana, Lesotho and Malawi, the paper has applications to 
developing countries elsewhere in Africa, Asia and Latin America. 



SECTION II 



EVALUATION PLANNING 

^To' plan IS to have an intention or purpose, and to form a scheme 
or method of achieving that intention or purpose. Planning involves a 
particular calculus of means and ends. Thus, evaluation planning, in 
the context of literacy campaigns, literacy programs or literacy projects, 
involves the development of an evaluation agenda and the scheme for 
implementing such an agenda. 

Planning Is a Socio-technical Process 

Evaluation planning (like all planning) is not a mere technical 
process. It is a socio-technical process. Evaluation plans and 
agendas prepared by professional evaluators must be put through the 
political process of two-tailed negotiation with policy makers, on the 
one hand, and with the beneficiaries of programs, on the other hand. 

This paper can deal only with half the process of evaluation 
pilanning the technical haTf. The other half of the process, which 
is a social process of negotiation, can not be conducted on paper. 
Indeed, it must be handled in real time and space, by real people^ 
projecting their different values and defending their diverse interests. 
That clearly we must leave" to the real stakeholders within the program 
system in each different country participating in this Workshop. 

Steps in Evaluation Planning 

The following four steps are involved in the process of evaluation 
planning: 

13 
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1 • Description of the "Evaluand" in Design Terms 

To plan is to choose. But to choose well, we must first know what 
to choose from. This means that we must have a sufficiently comprehen- 
sive description of the "evaluand" (the entity to be evaluated in 
this case a literacy campaign, a literacy program or a literacy project). 

Further, this description should not only be sufficiently compre- 
hensive, it should ]ay bare the hidden calculus of means and ends 
embedded within the evaluand. In other words, this description must be 
in design terms. This is important since the whole purpose of evalua- 
tion is to redesign the evaluand for better performance. 

What does It mean to describe sufficiently comprehensively and in 
design terms? This means to describe the evaluand in system terms: 
(i) to delineate the evaluand in terms of systems and subsystems involved 
and (ii) to sketch each of the subsystems in terms of the four systems 
parameters Contexts, Inputs, Processes, 'and Outputs (Products or 
Outcomes) . 

2. Using the Dynamic Description to Generate an Idea l-Type 
Set cf Information Needs and Evaluation Questions 

Once a description of the evaluand in design terms has been com- 
pleted, the various entries under contexts, inputs, processes and outputs 
should be used as triggers to thought, in permutations and combinations 
with each other, to generate an ideal-type set of information needs and 
evaluation questions. 

3- Developing a Program-Specific Evaluation Agenda 
From the ideal -type set of information needs and from the list of 
evaluation questions theoretically possible to ask, an evaluation agenda 

io 
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must be developed within the context of a particular program and its 
l-'^'^nTiation needs, at a particular moment in the history of that 
program. 

The two most important criteria 'jsed in the development of a 
specific evaluation agenda for a particular program will have to be: 

Significance . The evaluation questions chosen should be signifi- 
cant and not trivial. Having or not having the information sought by 
the evaluations should indeed make a difference. In other words, the 
evaluation topics chosen should be high-risK topics. 

Feasibility > The evaluation questions chosen should be feasible 
to study in terms of the availability of resources, the time frame 
within which results must be produced and in terms of the politics of 
the program at that time. 

4. Validation of the Evaluation Agenda Through Social Processes 
The evaluation agenda so developed should be justified, defended 
and, in the process, validated by putting it through the social process 
of two-tailed negotiation: on the one hand, with policy makers, program 
planners and other stakeholders within the system; and, on the other 
hand, with the beneficiaries of the program who will experience the 
program and who will provide most of the feedback and information on 
impact to the evaluators. 

Evaluation and Other Modes of Knowledge Production 

So far, we have bt^n using the word "evaluation" generally to cover 
all modes, methods and procedures of collecting feedback information and 
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knowledge about programs. In literature of evaluation, one will come 
across other words and phrases such as research, quick appraisals, 
monitoring and management information systems (MIS's). It is useful to 
understand the differences in the meanings of these various terms. 

Evaluation is the process of collecting and analyzing data about a 
program objectively, systematically and with exactitude with the purpose 
of serving the decision- makin g needs of the program . The evaluator's 
loyalties are to the program he or she seeks to improve. 

Research as a process is no different from evaluation. Both the 
evaluator and the researcher share inquiry designs, methodologies, tools 
and instruments and both have similar concerns about reliability and 
validity of their findings. The difference is in the choice of questions 
and in loyalties. Researchers have discipline-related academic 
orientation. They seek not to serve an on-going program but to advance 
the frontiers of knowledge in their chosen disciplines. Their loyalties 
are thus to a discipline (hot a program) and their choice of research 
topics is determined by the theoretical and research needs of their 
discipline rather than the decision-making needs of a program of action. 

Quick appraisals are quick evaluations, conducted under conditions 
of emergency to investigate the cause of a breakdown, to anticipate 
problems of implementation, or to get early returns on the impact of a 
program. 

Monitoring (or concurrent monitoring) involves watching or .checking 
upon an on-going program to detect flaws and to provide corrective 
actions . 
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Management information system is, as the phrase suggests, a system 
for collecting program information that will help in the management of 
the program. The need for having a management information system (MIS) 
is so obvious that one wonders why we took so long in our various manage- 
ment tasks to discover the necessity of an ;!TS. An MIS can — and 
should — serve as the steel frame around which the tasks of monitoring, 
appraising, evaluating and management are built. 

We should indicate here that the development of an MIS should always - 
be one aspect of evaluation planning within programs of adult literacy 
and other large-scale nonformal education activities. (See Section III 
below.) 

Program Planning and Implementation Versus 
Evaluation Planning and Implementation 

Ideally, evaluation planning should be part of program planning. 

Evaluation must be built into the programs so that progi^m planners can 

know the terrain of their work, can conduct effective soohbI diagnoses, 

can design appropriate interventions, and can collect systematic feedback 

for studying the impact of their programs on the lives of individuals 

and coimunities. 

Some implementation of evaluation must indeed precede the imple- 
mentation of the evaluand. That would be called context evaluation. The 
data collected at this scage of evaluation will describe the* cpntext in 
which the program will have to be implemented, will assess learner needs 
and will provide the base-line data for comparisons later on.^ ' 

Some implementation of the evaluation effort will be conducted , 
during the conduct of the program, dealing with program formation, 
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collecting information on Inputs (input evaluation) and on the processes 
of intervention (process evaluation). 

Another important part of the implementation of evaluation will be 
contingent upon the implementation of the program, and will be the 
evaluation of impact . One cannot talk of impact evaluation (or output 
evaluation) unless there is a program on the ground; and unless the 
program has had some time to take effect and show results in the lives 
of people and communities* 

In cases where a full-fledged program is not already on the ground, 
evaluation planning will have tOv.be mostly anticipatory. Evaluation 
planning will then become program planning. In suggesting what might be 
useful context evaluation and what might be useful feedback information, 
we will be talking about what might be useful to do in the first place 
and what kinds of literacy and post-literacy programs should be estab- 
lished and tested- in- use. , 

The Literacy System and Its Subsystems: 

Descriptions in Design Terms and Possible Evaluation Questions 

As was indicated in an earlier part of the paper, description in 
design terms involves a delineation of the evaluand into systems and 
subsystems and then descriptions of each of the systems and subsystems 
in terms of the four systems parameters — contexts, inputs, processes 
and outputs. In the following, we undertake this task of description. 

A delineation of the overall system of literacy (campaigns, pro- 
grams or projects) in terms of systems and subsystems is presented on 
next page. (See Figure II.l). 
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Figure II. 1: The many subsystems of the total literacy system placed 



within the overall development/societal system. 



ERIC 



20 

The literacy system should be seen as placed within and in inter- 
action with the overall development/societal system. The literacy 
system itself may be seen as being composed of the following sub- 
systems:^ 

1. The policy and planning subsystem 

2. The administrative and mobilizational subsystem 

3. The technical resources and instructional development subsystem 
4^ The teaching-learning subsystem at the community level 

5. The post- lite racy subsystem, and 

6. The evaluation subsystem. 

A couple of points need to be made here before proceeding further 
with Lhe discussion. A different carving of the subsystems within the 
overall system of literacy is possible. Another writer with somewhat 
different orientations might come up with a somewhat different list — 
not too different, however! The various subsystems will be composed, 
in turn, of sub-subsystems. Indeed, third-, fourth- and fifth-generation 
subsystems may be possible to define. It should be noted also that the 
evaluation subsystem itself could be evaluated — that would give us 
meta evaluation. 

In the following, we will now take each of the subsystems of the 
literacy system, one by one, to provide design-oriented descriptions for 
each subsystem, using the four system parameters of contexts, inputs, 

^This material is adapted from H.S. Bhola, Campaigning for 
Literacy (A Critical Analysis of Some Selected Literacy Campaigns of 
the 20th Century, with a f*temorandum to Decision Makers). Paris: 
Unesco, 1983. (See Chapter XII: Planning, Implementing and Evaluating 
Literacy Campaigns: A Memorandum to Decision-Makers). 

ERIC 
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processes and outputs. It is important to remember that these four 
categories are both necessary and sufficient. It is not. necessary to 
invent new categories, though each of these four could be broken down 
into smaller parts. For example, contexts could be organizational and 
social; inputs could be material and institutional; processes could be 
instructional and organizational; and outputs could be immediate and 
long-term. 

On the basis of these descriptions — thinking with them — we 
will suggest a set of possible evaluation questions of practical signifi- 
cance. While these questions will, we hope, constitute useful sugges- 
tions, readers/participants should not take them as prescriptions. They 
may come up with additional questions or alternative questions, within 
the contexts of their particular programs. *^hen they niust develop 
evaluation agendas to work on during the next few months or years. 

1 . The Policy and Planning Subsystem 

The policy and planning subsystem is perhaps the most important 
subsystem within the overall system of literacy. This is a subsystem 
which must undertake to establish relationships between the literacy 
system and the developmental priorities of the nation (that is, with 
the development/societal system). It must define the temper of the 
literacy effort: Will it be a reconstructionist strategy for bringing 
about a new political culture; or will it be construed as a develop- 
mental strategy for improving productivity and participation within the 
existing social and economic order? 

The policy and planning subsystem must also make decisions about 
generating and allocating resources for the literacy effort. How much 
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will becormiitted to the literacy effort? Where will these resources 
come from? Will the people be expected to cormiit their own personal 
resources to the literacy effort? 

This subsystem wst also make decisions about the kinds of 
institutional resources which will be created or comnitted to literacy; 
coordination that will be encouraged or enforced; and the mobilizational 
strategies that will be permitted to be used to recruit people as 
learners and teachers. It must also decide upon the scope and style 
of people's participation in the overall literacy program. Finally, 
the role of evaluation itself must be established within the literacy 
system and nature of accountabilities at various locations and levels 
must be clarified. 

In the table placed on next page, we have described the policy 
and planning subsystem in design terms. Thinking with this tabulation, 
we can suggest, by way of example, the following evaluation questions;^ 

(i) What inspirations can be drawn from the country's historical 
heritage and experience so as to generate policy justifications that 
appeal to the masses? Was the written word given a sacred definition 
in the tradition (as was done, for instance, in India)? Or, if the 



It should be noted that many of these so-called evaluation ques- 
tions could be better characterized as program planning questions. As 
was indicated earlier, questions of '^context evaluation" are indeed 
equivalent to those for program planning. When a program is yet not on 
the ground and evaluation planning is anticipatory, the line between 
evaluation planning and program planning becomes very thin. It should 
also be noted that one would not have to go to the field to collect 
erppirical data to answer aV[ evaluation questions. Many of the most 
significant evaluation questions may have to be answered using logical 
and documentary analyses, sitting in one's office. 

2o 
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Table ILl: The Policy and Planning Subsystem 



CONTEXTS 



!• Historical antece- 
dents 

2. Current historical 
conditions: 

2.1 Ethnic/racial 
divisions 

2.2 Language 
divisions 

2.3 Class 
divisions 

3. Existing political 
culture 

4* Geographical/ 
demographic 
context 



INPUTS 



!• Policy makers and 
pi anners 

1 . 1 Their experi- 
ence and tools 

2. Ideology/national 
visions 

3. The masses 

3* 1 Collective 
aspirations 

4. Social and 
economic informa- 
tion 

5. Institutional and 
technological 
infrastructures 

6. Resources 



PROCESSES 



Political 
1 . Of leadership 
recruitment 

2- Mobilizational/ 
communicational 

3. Of incorporation/ 
affiliation 

4. Of validation 

Technical 
5^ Cognitive/ 

decision 

model 1 ing 

6. Institution 
building 

?• Of administration 
and coordination 



OUTPUTS 



1 . Policies and 
plans 

2. Institutional 
structures to 
support, deliver,^ 
and review implenien- 
tation of proposed 
actions 

3. Language of 
policy 

Justification in 
terms of rural 
development 

4. Dissemination of 
policy statements 
and plans 
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culture was an oral culture, are there myths and legends that extolled 
knowledge and can be used to promote adult literacy and the acqiisi- 
tion of knowledge? 

(ii) Was literacy promotion a part of the independence movement or 
another revi tal ization movement of some kind? What part did literacy 
play in such a movement? What does this experience teach us in regard 
to future planning for literacy promotion? 

(iii) What, if any, are the ethnic, racial and clas<; bases of 
literacy and illiteracy? 

(iv) What language or languages have the potential to become the 
language(s) of literacy in the country? What languages are spoken by 
how many? What languages are understood by how many? What is the 
current and potential economic and political role of each language?^ 

(v) What institutional arrangements and mobilizational strategies 
are possible within the existing political culture? 

What special strategies of communication and mass involvement 
have been and can be possible in the context of the existing geographical 
and demographic realities? 

(vii) What are« the expressed aspirations of the peoples? What is 
their level of commitment to the official ideology? What are their per- 
ceptions of the need of adult literacy? 

(viii) What are the profiles of available resources, existing 
infrastructures, and soc;ial and economic indicators? 



It is quite likely that evaluators working on a government 
sponsored literacy program will take government plans and policies for 
granted and will not want to raise many of the questions listed in this 
section. 
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(ix) What is the nature of the current traditional leadership 
within connunities? What roles are being performed by such leadership? 
Can this traditional leadership be recruited to perform the modern task 
of eradicating illiteracy? What might be the effects of introduci.<g 
new secular leadership within existing traditional structures? 

(x) What will be the process and effects of the integration of 
the new literates into the economic, social and political structures? 

(xi) What processes of validation of po.licies, plans and strate- 
gies are possible with and are preferred by the beneficiaries of 
programs? 

(xii) What level of inter-agency coordination, as envisaged by 
policy makers, has been possible? Where are the breakdowns? 

(xiii) Are the new institutional mechanisms established under 
new policy arrangements performing their tasks? 

(xiv) Are the literacy related policies approved and appreciated 
by different constituencies involved? 

Z. The Administrative and Mobil izational Subsystem 

Policy initiatives once established must be implemented through 
appropriate administrative actions; and popular involvement must be 
created through mobilization. The Burmese, in their literacy 
campaign of 1960s, talke^* of "organizational power" which was able to 
put the scarce resources of the state at the service of the literacy 
campaign, but more importantly was able to mobilize resources which 
the government did not have, by enthusing people to give of and from 
themselves. 
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The Contexts-Inputs-Processes-Outputs analysis of the administra- 
tive and oiobilizational subsystem is presented on next page. (See 
Table IK2) Since policy making (already discussed above) involves 
the dual process of establishing directions and harnessi ng of institu- 
tional ^esoujrces for implefnenting new directions, some of the entries on 
the following tabulation will be similar to the entries in the tabula- 
tion for the "Policy and Planning Subsystem." (Table II, 1 above) 

Thinking with the tabulation II-2, one can generate the following 
list of evaluation questions: 

(i) What are the value profiles of the development bureaucracy at the 
various levels of the nierarchy? Are those value profiles congruent 
with the task of eradicating illiteracy? 

(ii) What are the educational backgrounds and staff development needs 
of decision makers within the administrative system? 

(iii) What is the organizational capacity of the administrative 
system and of the voluntary and mass organizations for the delivery 
of literacy? 

(iv) What are the social -psychological make-up, political orientation 
and pedagogical abilities jf cadres working in the literacy program? 

(v) What are some of the indigeneous structures of cornmuni cation and 
leadership which can be used to mobilize the people? What linkages 
are possible between traditional and modern media and modes of inter- 
action? 

(vi) How are the new roles (of literacy teachers, supervisors, etc.) 
introduced into the cofnmunities working at the community levels? Are 
role expectations being fulfilled? Are there any role conflicts emerging 



2< 



e IK 2: The Administrative and Mobi lizational Subsystem 



CONTEXTS 



INPUTS 



PROCESSES 



OUTPUTS 



Policy making 
culture 

Bureaucratic 
culture 

Social ecology 



1. Development bureau- 
cracy 

1 >1 Their values 

T72 Tfieir training 

2. Organizational/ 
institutional infra- 
structure 

2.1 Organizational 
capacity 

2.2 Tools and 
technology 

3. Mass organizations 
3. 1 Missions, 

capacities 

4. Voluntary organiza- 
tions 

4/1 Missions , 
capacities 

5. Agents of government; 
Cadres of political 
parties and mass 
organizations at the 
field level 

6. Communication struc- 
tures for informa- 
tion dissemination 
and mobilization 



Administrative 
1 . Task analysis and 
inventorying 



2. Role design 

3. Organizational 
design 

4. Organizational 
development (OD) 
and communication 

5. Staff development 
and team building 

6. Coordination, 
interfacing, 
linking 

7. Management 
(Perting, 
Scheduling, & 
Budgeting) 

Mobi lizationa l 

8. Communication 

9. Motivating 

10. Building commitment 

11. Eliciting response 



1. Organizations 

2. Roles 

3. Coordination 
mechanisms 

4. Adhocratic committees 

5. Mobilization 
mechanisms 

6. Message systems 
(Slogans, oaths, 
posters, folk media 
and radio programs, 
etc.) 

7. Public involvement 
and response 

8. Community action 
and commitment of 
public resources 
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in relation to other roles and expectations? 

(vii) What is the perception of the beneficiaries about the program: 

a program in the Interest of the State and under government control, or 
a program in the interest of people enjoying full support of the 
-government? 

(viii) Is the program actually integrated at the point of delivery? 
Is the program seen as one unified service or is it experienced as too 
many government functionaries getting into each other's ways? 

(ix) What does an analysis of people's responses (the social graffiti 
of slogans, writing on the walls, posters and anecdotes) tell us about 
how the people are experiencing and responding to the program? 

(x) What does an inventory of community actions, following the literacy 
program, look like? 

3. The Technical Resources and Instructional Development Subsystem 

This, again, is an important subsystem. Instructional Development 
Cthe process of develpping instructional packages based upon task 

3 

analysis, learner analysis and analysis of learners' environment) is 
the heart of the system of literacy. Indeed, writing of primers and 
preparation of posters, drill cards for learners and development of 
teacher guides for instructors are sometimes seen as tlie problem of 
literacy promotion. (This narrow conception of literacy work is 
unfortunate, but it, nonetheless, points up the great significance of 
the subsystem of instructional development and related problem of mate- 
rials production for literacy work.) 

Related with the task of instructional development is the task of 
building technical resource systems to support the technology of literacy 

\ 
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work- This means creation of institutions which can undertake a whole 
array of RD&D (Research-Development-Dissemination) activities in support 
of literacy work. India is, perhaps, the only country which has 
systematically and carefully established a network of resource centers 
from the center to the states (and hoping to take it down to the level 
of districts) in support ot their literacy effort as part of the nation 
wide Adult Literacy Program launched in 1979. 

A Contexts-Inputs-Processes-Outputs description of the subsystem 
(which indeed it composed of two sub-subsystems: the instructional 
development sub-subsystem and the technical resources sub- subsystem) 
is presented on the next page. (See Table II. 3) Once again, thinking 
with the tabulation one can raise literally hundreds of questions in 
this subsystem. The list of evaluation questions presented below is 
l^no means exhaustive but does suggest some significant possibilities: 

(i) Do the prevalent political doctrines demand one centralized 
curriculum and one set of materials or does it encourage one doctrine 
but decentralized approaches to the promulgation of the doctrine? 

(ii) Does the political culture permit the development of genuinely 
competitive doctrines and thereby competitive instructional systems 
and content of instruction? 

(iii) What is the latent curriculum of the materials that make part of 
the instructional system? Does the approach involve political resocial- 
ization? Does it involve work-orientation? 

(iv) What resources are available at national and regional levels in 
regard to (a) social change planning and pedagogy; (b) writing creative 
and expository materials; and (c) artists, photographers, printers, film 
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Table II .3: The Technical Resources and Instructional Development Subsystem 



CONTEXTS 



INPUTS 



PROCESSES 



OUTPUTS 



L Political context 

1.1 Conformist versus 
pluralistic culture 

1.2 Right to hold and 
promote alternative 
ideologies and 
message systems 

1.3 State's control 
of institutional 
resources or 
otherwise 

2. Intellectual /Educational 
context or climate 

2.1 Availability of 
writers, artists, 
media' specialists , 
pedagogues 

2.2 Pedagogical 
doctrines 

3. Technological infra- 
structures 

3.1 Paper-making 
factories 

3.2 Printing facil ities 



Manpower: 
researchers , 
developers , 
program 
specialists , 
literacy 
workers 

Institutions: 
universities , 
research 
centers , 
libraries , 
publishers 

Materials 
resources : 
printing and 
photographic 
equipment , 
paper, 
other 
supplies 



1. R,D&D 

2. Institution 
building 

2.1 Staff 
development 

2.2 Training 

3. Instructional 
development 

3.1 Task, 
learner, 
and environ- 
ment analysis 

3.2 Design, 
production, 
and delivery 
of instruc- 
tional 
products 



1 . Instructional 
products : primers, 
work books, print 
and audiovisual 
materials, gui de- 
books ^ followup books 

2. Trained manpower: 
teachers, group 
leaders, monitors, 
supervisors 

3. Instructional 
systems: media 
systems, distance 
education, f 
correspondence I 
learning 

4. RD&D Institutions: 
Research centers, 
publication channels, 
learning resource 
centers 
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makers, radio and television producers? 

(v) What is the profile of available manpower of direct concern to the 
task of literacy promotion? What are their numbers? What is the quality? 

(vi) What is the existing level of commitment of support institutions 
such as universities and research centers to literacy work as demon- 
strated in their training programs and research outputs? 

(vii) What are the titles published by private publishers during the 
last decade qn the subject of needs and methods of literacy promotion? 

(viii) Are any courses being taught at the universities on the subjects 
of literacy and adult education? What is the content and quality of 
these courses? What is their annual output? 

(ix) What is the quantity and quality of material resources available 
for instructional developers and technical resource builders to accomplish 
their tasks? 

(x) Is the instructional development process based, on the one hand, 
on appropriate needs assessment procedures and, on the other hand, on 
appropriate analyses of tasks, learner characteristics, and environmental 
variables? 

(xi) Do instructional materials reflect innovative design and high 
quality of production? 

(xii) Does the instructional development strategy accommodate alter- 
native conceptions of content and method of teaching literacy? 

(xiii) Does the instructional packaging of the materials allow partial 
unpackaging and local adaptation of materials? 

(xiv) Are there opportunities for peer reviews and professional audit 
of instructional materials produced? 
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(xv) Have the procedures of instructional development been institution- 
alized enough for the process to continue at least during the life of 
the project? 

(xvi) Have appropriate support institutions been created and inter- 
faces established with universities and research centers to complete the 
chain of the RD&D process? 

(xvii) What will be the logistical requirements of a system to be aole 
to deliver materials and instructional services to those who need them 
and where they need them? 

(xviii) Are primers and related reading materials based on the right 
vocabulary research? Are they properly designed and graded? Do they 
teach effectively? ' • 

(xix) What -are the levels of performance of teachers, group leaders 
and supervisors in the system as seen by program specialists and 
beneficiaries? 

(xx) What are the levels of organizational capacity and performance 
levels of the institutions of instruction and instructional support? 

4. The Teaching-Learning Subsystem at the Community Level 

This subsystem encompasses instruction and development action as a 
living system. Learners and teachers now make actual instructional 
transactions. Learners transfer their learning to others and to their 
life and work, to take actions that translate as change at the community 
level and at the national level. 

A Contexts-Inouts-Processes-Outputs description of the subsystem is 
included on tne following page. (See Table II. 4) Thinking with this 
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Table II. 4: Tlie Teaching-Learning Subsystem at the Community Level 



CONTEXTS 



INPUTS 



PROCESSES 



OUTPUTS 



1. Cultural 
(especially 
religious) and 
demographic 
context 

2. Ethnic/Caste 
organization of 
the community 

3. Social/ 
Geographical 
ecology 

4. Economic organi- 
zation and/or 
occupational 
profile 

5. Community experi- 
ence with earlier 
literacy and 
development 
actions 



1. Learners 

1.1 Families 

1 .2 Salient 
others 

2. Teachers/ 
Cadres/ 

Extension agents 

3. Instructional 
materials 



3.1 



3.2 



Community 

learning 

resources 

Community 

leaders 



1. Communicational 

2. Organizational 

3. Instructional 

4. Socializational 

5. Of linkage and 
transfer 



1. Individual learning/ 
productivity/ 
resocialization/ 
praxis 

2. Family and community 
learning 

3. Effective 
traditional/modern 
leadership 

4. Community action 
for development 

4.1 New learning 
structures (skills 
exchanges; community 
centers; LRC's) 

4.2 New economic 
structures 

4.3 New social 
structures 

4.4 New political 
structures 
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tabulation, a set of evaluation questions ^can be generated as follows: 

(i) What does the cultural and demographic context of field work 
suggest about grouping of learners — independent learners, teacher- 
learner dyads, families, peer groups, age groups, mixed groups of males 
and females, and separate classes for men and women? How does the 
context affect choice and location of learning sites? 

(ii) What does the economic organization of the community, and the dis- 
tribution of occupations within the community, suggest in regard to the 
"functional" content of the program in the field? 

(iii) What are the perceptions of the community about the objectives, 
procedures and impact of some development programs offered to them in 
recent years? Do the potential beneficiaries of the proposed literacy 
programs see these programs as particularly helpful? 

Civ) Who is in the program? For what reason? For how long? With 
what effect on himself or herself and on their families? 

(v) What are the class orf^ins, values, and instructional and conimunica- 
tiph skills of those engaged in the program as teachers, monitors and 
grdup leaders? 

(vi) ' Does coordination .between and among various functionaries exist 
for as integration of "words" and "work" to be actualized? 

(vii) Are indigeneous learning resources (knowledgable members of the 
Cv.n¥nunity, folk media and traditional institutions, for instance) being 
put to work and how? 

Cviii) What channels and modes of instruction, communication and 
organization are working inadequately and what are working particularly 
well? Why? What is the special role assigned to radio. in the teaching 
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of adult learners in classes and the community? 

(ix) Is adult learning going beyond reading and writing and farther 
than mere acquisition of knowledge? Is literacy being put to use? Is 
there evidence of behavioral change? 

(x) Is learning first achieved by the primary group of learners 
spreading or filtering through to the community? How and what is being 
transferred? 

(xi) What is the level of acquisition of literacy, numeracy and func- 
tional economic knowledge among the adult learners in the program? 

(xii) Have there been increases in the levels of productivity? What 
uses are being made of the increased disposable income? 

(xiii) Has literacy contributed to political participation on the part 
of adult literates? How and in what measure? 

(xiv) Have there been changes in the leadership patterns at the 
community level? What are those changes? 

(xv) What kinds of pressures have been generated that effect the 
institutional life of the community? 

5. The Post-Literacy Subsystem ^ 

The post-liter^acy subsystem is itself composed of at least four 
sub-subsystans and is by no means a simple concept to understand. 
Before presenting the Contexts-Inputs-Processes-Outputs table for post- 
literacy, we will present some introductory material on the concept 

^Material presented in this section was part of the paper, "Evalu- 
ation Planning of Post-Literacy Programs" presented at the National 
Seminar on Strategies for Post-Literacy, Followup and Continuing Edu ca- 
tion in Rural and Urban Context , organized jointly by the Directorate "of 
Adult Education, Ministry of Education and Culture, Government of India 
and State Resources Center at Osmania University, Hyderabad, India, 
during September 24-30, 1982. 
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and enterprise of post-literacy. 

The following remarks should be explanatory: 

Posf-Literacv: the Time Frame 
and the Program Frame 

Post-literacy as a concept is a mix of the temporal (the time 
dimsnsion) and of the programmatic (the program dimension). Post- 
literacy in its program dimension consists of programs and projects that 
are built upon the assumption of literacy (or at least without the 
assumption of illiteracy) among those sought to be served by post- 
literacy programs. Thus, illiteracy is not the criterion used for the 
choice of modes or media of instruction in post-literacy programs, even 
though these programs m^ often serve both the literate and the illiterate. 
Paradoxically, post-literacy programs are not always for the already 
literate. 

In their temporal dimension, post-literacy programs should follow 
literacy programs in time. This is typically so when seen from the 
perspective of the participants of one particular literacy program. But 
time is relative from the perspective of planners of post-literacy pro- 
grams. Planning of literacy and post- literacy programs can and indeed 
must be handled by them concurrently and conjointly. The implementation 
of literacy and post-literacy programs will also be often concurrent 
because while some client groups will need initial literacy instruction, 
sorie others will need , post-literacy programs, having acquired their 
literacy skills in earlier adult literacy programs, or within the context 
of the school. 
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Clients of Post-Literacy Program 

Unfortunately, the word post-literacy has often encouraged a 
fixation on the needs of adults coming out of liceracy classes organized 
for them in out-of-school settings. Thus, only those coming out of 
adult literacy classes have been seen to be the clients of post-literacy 
programs. This is a narrow conception and planner of post-literacy 
programs must grow out of it. 

Post- literacy programs should be planned to serve all those who are 
literate, irrespective of the setting in which literacy was first 
acquired by the now literate. This means that clients of post- literacy 
programs will include adult men and women coming out of literacy 
classes especially organized for them; school leavers and dropouts from 
school, with various levels of literacy acquisition; and any others who 
my have acquired initial literacy at home, at work, in a religious 
establishment or in the arny. Some may be self-taught. 

Four General Objectives of 
Post-Literacy Programs 

Four general objectives of post-literacy programs can be identified 
(i) Li teracy -Retention : This involves reinforcement and strengthening 
of literacy skills of new literates through use, making relapse into 
illiteracy unlikely. Sometimes there may be a prior task, that of 
remediation, when the level of literacy acquisition among participants 
is found not to be of a sufficient level. 

^Bhola, H.S. (with Joginder K. Bhola). Program and Curricul um 
Development in the Post-Literacy Stages . (A Workshop Manual), Bonn, 
FRG: German Foundation for International Development, 1980, p. 16. 
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(ii) Second Chance Formal Education : The objectives of post-literacy 
may often involve providing a second chance for formal primary or 
secondary education to those who had missed the chance of going to school 
when young* or are unable to go to school now because there is no school 
to go to in the community. This can be called Alternative Formal Educa- 
tion or AFE* 

tili) Systemic Integration : The now literate are taught additional 
literacy skills and functional knowledge with the objective of helping 
them to integrate within the social, economic and political institutions 
and structures of the society^ to the mutual advantage of both the 
individual and the social system. 



(iy) Socialization for an Ideal Society : The now literate are prepared 
for creating,, sustaining and for participation in an ideal community and 
for contributions to a new world order. 

The figure below (Figure II. 2) shows the four sub-subsystems of 
i post-literacy in relation to the various systems of education and 
instruction that feed 'the post-literacy subsystem. 

It is now time to present a Contexts-Inputs-Processes-Outputs 
tabulation for the post-literacy subsystem. As could be judged from 
the figure below, the tabulation would indeed be four tabulations in 
one as shown on page 40. (See Table II. 5) 
' The entires in the tabulation ^a re not meant to be comprehensive 
for reasons of space and scope of this working document. But they 
are, nonetheless, indicative of the usefulness of the tabulation to 
assist in the generation of a -series of appropriate evaluation 
questions. ^ 
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-Societal system 

Figure IL2: The various components of post-literacy and the various 
systems of education and instruction feeding the subsystem of post- 
literacy, 
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Table II. 5: The Post-Literacy Subsystem 



CONTEXTS 



INPUTS 



PROCESSES 



OUTPUTS 



1. Literacy 
program 
>. o context 



LU LU 



2. General social 
context 



i-H LU 



1 . New literates 

2. Instructors 

3. Writers 

4. Reading materials 

5. Institutions for 
promotion of 
reading 



1. Research 

2. Writing, testing 

3. Publication 

4. Instruction 

5. Organization 

6. Distribution 
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Formal educational 
context 

General socio- 
economic context 



o 

IJLj CO 

H- UJ 
00 I- 
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Social , economic 
and political 
structures 

Political 
culture 



1. Cultural context 

2. Global realitias 



^ LU 

UJ o 

o o 

M-i 00 



1 . New literates 5 1 . 
dropouts, school- 
leavers 2. 

2. Teachers, instruc- 
tional developers 3. 

3. Materials 4. 

4. Support institutions 5. 



Instructional 

development 

Publication, 

distribution 

Instruction 

Organization 

Examining 



1 . Learners 

2. Extension agents 

3. Instructional mate- 
rials and training 
tools for produc- 
tivity 

4. Support institu- 
tions 



1. Instructional 
development 

2. Teaching and 
training 

3. Economic and 
political 
affiliation 

4. Institution 
building 



1 . Learners 

2. Change agents 

3. Materials 

4. Support institu- 
tions 



1 . Instruction 

2. Consciousness 
raising — '"^i 



1. Literacy retention 

2. Literacy utilization 

3. A reading public 

4. Institutions for 
writing, publishing, 
distributing books 



1. Certified graduates 

2. Trained manpower 

3. Personal satisfactions 



1 . Economic, social and 
political democracy 

2. Responsive institu- 
tions 



1 . Individual and social 
praxis 

2. Appropriate social, 
economic and political 
agendas 

3. New world order 
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Some of the evaluation questions suggested by the table are given 
below: 

I. Some Questions at the Interface J 

(i) Is the instructional time assignejd to the literacy cycle 
sufficient to make adults in literacy classes literate? What about 
functionality? What about awareness? 

(ii) What is the level of literacy acquisition of dropouts and 
school leavers coming out of elementary schooisf 

(iii) What is the age, sex, occupation, income and class composi- 
tion of those declared literate through adult literacy programs? 

(iv) What is the new literate's view of the illiterate? What is 
the illiterate's view of the new literate? 

(v) Is there a difference between the literate and the illiterate 
in regard to their knowledge of "how the world works"? 

(vi) What is the experience in reaching the "poorest of the poor" 
and other disadvantaged groups during the literacy program which could 
now be used to reach them in the post-Jiteracy stages? 

(vii) What are some typical profiles of aspirations of peoples — 
illiterate, new literate, young, old, male, female? 

I I . Literacy Retention: Some Evaluation Questions 

Some of these questions may be better characterized as research 
questions but they all have practical significance in relation to the 
design and evaluation of a post-literacy program: 
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•1 The Phenomenon of Retention/ ^ 

(i) How many of those declared literate typically lose their 
literacy skills and under what/condi tions? 

(ii) Who retains literacy? What level of literacy makes a person 
retention-prone? What are the motivations of those who are able to 
retain literacy? What are the uses to which they put their skills? What 
is their work and life context? 

•2 Provision of Reading Materials 

(i) How have the prize schemes, if any, dealing with the production 
of reading materials done? What is the natural history of a prize 
winning book? How do the sales of prize winning books compare with 
those that did not win prizes? In other words, what is the economics 

of prize publications? 

(ii) Do prize winning publishers later publish more books for new 
literates? 

(iii) What is the level of productivity of prize winning authors? 
Do they write more books for new literates? 

(iv) Do nationally known popular writers want to write for the 
new readers? Why? Why not? 

(v) Is writers workshop a useful format for training writers of 
post-literacy materials? 

(vi) Is writers workshop a useful format for actual production of 
materials for new readers? 

(vii) How do different models of writers workshops in use in the 
country and abroad compare with each other in terms of productivity and 
econony, efficiency and effectiveness? 
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(viii) When can writers cooperatives succeed in the provision of 
post-literacy materials for new readers? 

(ix) What has been the experience with translation of books for 
new readers from one language into another? What has been the experi- 
ence with co-publishing, if any? 

(x) What has been the experiencH'lJTtfrth^^ of available 
materials (such as extension materials in agriculture, health, nutrition 
and family planning) to the special needs of the new literates""OTd--now — 
literates? 

(xi) What has been the experience with readers-written books? Or, 
with the writing of books based on the oral history of a community?^ 

•3 Vocabulary Research and Evaluation 

(i) What is the spoken vdcabu-lary of the prospective participants 
in programs of post-literacy? 

(ii) What written words have become sight words for those coming 
out of literacy classes and for those coming out of schools? 

(iii) Can we taxonomize levels and contents of literacy in terms 
of specialized word lists? 

(iv) What words must be mastered by learners to cope with 
scientific agriculture, family planning information, membership in 
cooperatives and such? 

(v) What factors should enter into the design of readability 

^Bhola, H.S* Writing for New Readers: A Book on Follow-up Books > 
Bonn, FRG: German Foundation for International Development, 1981, 
190 pages. 
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formulas for measuring the readability levels of reading materials in 
languages chosen as languages of literacy? 

(vi) What are the reading interests of adult men, adult women, 
and youth as expressed by them? What reading materials are actually 
chosen by them under conditions of free choice? 

Cvii) How to do content planning of books for new readers to relate 
them to their information needs on the one hand and their existing world- 
views on the other hand? 

(viii) What is the nature of good writing for new readers as dis- 
covered from the utilization of books already written for them? 

(ix) What are the special considerations in regard to type size, 
illustrations, book size and binding for the literature for new literates? 

(x) What might be/the special features of the so-called problem- 
oriented reading materials? 

(xi) What might be the special features of materials written for 
reading aloud by a literate to a group of illiterates? 

(xii) What format(s) might be selected for post-literacy and 
extension materials so that these are not seen as ephemeral but can be 
stored and filed for later reference by reading circles? 

(xiii) How to go about field testing of reading materials? Can 
this process be routinized? 

•4 Delivery of Services and Organizational Aspects of 
Post-Literacy Programs Dealing with Retention 

(i) What are the comparative advantages and disadvantages of 
various models for the delivery of post-literacy such as organized post- 
literacy groups, mobile libraries, learning posts, distance education. 
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etc.? 

(ii) What are the advantages and disadv^antages in the operation 
of supervised post-literacy programs (such as programs offered in 
organized groups with trained tutors) and independent post-literacy 
programs (where learners make independent decisions to borrow reading 
materials) . 

(iil) What are the profiles of good libraries? What are the 
problems of unsuccessful libraries? 

•5 Recruitment and Training of Cadres 

(i) What are the special qualifications and characteristics, and 
training needs of post-literacy workers? 

(ii) Can literacy teachers become good organizers of post-literacy 
programs? 

(iii) JCan^teachers of post-literacy programs be mobilized through 
political work or do they have to be more formally recruited? 

•6 Habits of Media Consumption and Other Effects 

(i) What do people do with w!iat they read? 

(ii) Kr;at are the reading habits of new literates and now 
literates in regard to the use of rural newspapers and special magazines 
written for them? 

(iii) Does literacy and reinforcement of literacy lead to a new 
"technology of intellect"? When does the new way of classifying, 
reasoning and remembering take hold? 

(iv) Can reading be encouraged by serializing and clustering books 
and asking people to "Read a Hundred Books" or to "Read a Thousand 
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Books" as^they have tried in China? 

(v) Do literates in post-literacy programs have habits of informa- 
tion consumption different from those who never enrolled in such programs? 

(vi) How does the utilization of radio and TV differ among groups 
of illiterates, new literates, and now literates enrolled in post- 
literacy programs? 

(vii) How does literacy and post-literacy effect the relationships 
of literates with traditional media such as attendance at end enjoyment 
of puppet shows and the like? 

(vili) Does literacy and post-literacy change the relationship 
between the new literate and the power figures in his or her environ- 
ment, such as, the village headman, policeman, extension workers, govern- 
ment officials, and, in the case of women, men in general? 

IIL Some Evaluation Questions on 

Second Chance Education Programs 

Evaluation questions in this particular program stream of post- 
literacy can deal with the characteristics of clients of these programs, 
with curriculum, delivery of instruction, problems of legitimization of 
second chance education and effects of second chance education on 
graduates and on competitive educational systems: 

(i) What is the demographic profile of those joining or wanting 
to join second chance education programs? What are their future 
aspirations? 

(ii) What are the special needs of participants of second chance 
education programs? What are the problems of creating programs that 
are different in content but equivalent in objectives at various levels 
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of formal education?^ 

(iii) What are the comparative cost-effectiveness ratios for 
different modes of delivery for second chance edi.:=»tion: evening 
schools, correspondence education, accelerated day school? 

(iv) What is the legitimization given to graduates of second chance 
education programs within the social and economic system? 

(v) What are the effects of second chance education on partial- 
pants in relation to personal satisfactions, sel f-concepts , educational 
mobility, vocational success and increments in income, social prestige 
and political participation, etc.? 

(vi) What are the effects of second chance education programs on 
competitive systems of formal education and en the democratization of 
oducation in general? 

IV. Some Evaluation Questions on the 
Systemic Integration Theme 

Some general questions can be askediabout the very concept of 
integration in operation , in other words, on the operational meaning 
of integration; on the special kinds of cadres needed for effecting 
systemic integrations and the kinds of values needed among cadres for 
them to work towards such integrations. There is information needed on 
the economic and political opportunities available, on the problems of 
codification of information needed for more effective performance of 
institutions and the effects of integration on new literates and on 
existing institutions. Here are some examples of questions: 

(i) What do post-literacy workers do when they claim to work on 
the task of integrating new literates into the social, economic and 
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political structures of the society? 

(ii) What are the characteristics, qualifications and value pro- 
files of those working in programs of systemic integration? Are they 
aware of being engaged in tasks which are basically reconstructionist 
•and egalitarian? Are their values congruent with the tasks they have 
been recruited to perform? Can civil servants or apolitical cadres 
accp^lish the ta§ks of systemic integration? 

(iii) What is the range of institutions — traditional and 
secular — in which there are opportunities for new literates to 
participate? 

(iv) What has been the success of polyvalent adult education 
centers in urban areas in economic integration of their participants? 

(v) What are the possibilities of codification in writing of 
information and instruction for each available institution? In what way 
can written materials help in the capacitation of individuals for better 
role performance? How does the situation differ in regard to tradi- 
tional versus modc-rn institutions? Are some or most secular and modern 
institutions also working on the assumption of illiteracy among their 
members and clients? 

(vi) Do those who undergo post-literacy programs connect better 
with institutions in the society and demonstrate better integration? 

(vii) Do post-literacy programs seeking to promote systemic 
integration change the institutions themselves in regard to their leader- 
ship patterns and functional effectiveness and overall responsiveness? 
Are disaclvantaged groups participating in leadership and holding 
instructional roles within these institutions? 
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(viii) Does post-literacy result in dysfunctional adaptation, 
making it impossible for new literates to continue to work in the 
informal econor;i1c sev^tor in rural areas and forcing migration to the 
city to join the formal economic sector? 

(ix) Is political awareness actually resulting from post-literacy 
programs? With what consequences for new literates themselves and for 
the advan-taged classes? 

(x) What are the effects in terms of a general response to 
modernization of teaching scientific literacy to adults? 

(xi) What are the effects on high school graduates and university 
students of their participation in some national service scheme in 
regard to their political resocialization? 

(xii) What kinds of programs of systemic integration invite 
greater community participation? 

V. Socialization for an Ideal Society 

Many of the information needs for this stream of post-literacy 
prograifiS will have been met through the questions already raised. A few 
more questions are added here: 

(i) Do post-literacy programs make independent learners or is 
there a contradiction in establishing supervised groups and in offering 
organized programs to adults whom we want to become independent learners? 

(ii) Can one identify a general propensity or inclination anwng 
adults in post-literacy programs or graduates of such programs to engage 
in individual and social praxis? 

(iii) Is there, among the relevant groups of adults, evidence of 
media appreciation, of understanding technology, of ecological and 
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environmental concerns agnd of the need for a new international order 
based on peace and brotherhood of man? 

(iv) Have participants in post-literacy programs become better 
consumers of information and culture? 

(v) What are the possibilities of establishing "enabling" insti- 
tutions that will promote socialization of individuals for an ideal 
society? 

(vi) What happens to those who conduct post-1 itercicy programs in 
the field as instructors? Do they join the new secular leadership in 
rural and urban communities? 

^ 6. The Evaluation Subsystem 

Finally, we have the evaluation subsystem which is part of the 
overall system of literacy. As in the case of other subsystems, one 
can describe the evaluation subsystem as well in tern,: of the Contexts- 
Inputs-Processes-Outputs categories and ask evaluation quertions about 
the evaluation subsystem itself. This "evaluation of evaluation" is 
sometimes referred to as Meta Evaluation. Thinking with the tabulation 
on next page (see Table II. 6), one can generate the following types of 
questions about evaluation: 

(i) Does the policy making culture provide an environment where 
evaluation studies can be systematically conducted without outside 
political interference and without the fear of punishment for producing 
unfavorable results? 

(ii) Is the organizational structure and climate of the program or 
project amenable to conducting effective evaluation and to the proper 
utilization of evaluation findings? 



Table IL6: The Evaluation Subsystem 



CONTEXTS 



INPUTS 



PROCESSES 



OUTPUTS 



K The program context 

2. The policy making 
culture 

3. Knowledge base 

4. The technological 
infrastructure 



1 . Informati on needs 
of decision 
making 

2. Evaluators, 
inves:ti5it^rs 

3. Subjectsi^ 

4. Technology 

4.1 Hardware 

4.2 Software 

5. Material resources 

6. Support 
institutions , 



1. Evaluation 
planning 

2. Negotiation 

2.1 With 

poli ticians 

2.2 With 
communities 

3. Evaluation manage- 
ment 

4. Utilization of 
evaluation results 

5. Intervention in 
programs for 
improvement 



1. Data base 

2. Evaluation studies 

3. Staff development 

4. Improved programs 

5. Inst • tutions of 
evaluation and 
accountabi lity 
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(iii) Does an inventory of the information needed for servicing 
important program decisions exist? Has this inventory been systematic- 
al ly developed? 

(iv) What are the training needs of the evaluators at various levels 
of responsibility? What administrative, logistical and professional 
support do they need to be able to implement evaluation studies? 

(v) What are the social, economic and cultural realities that surround 
the subjects of evaluation who must be observed and questioned for 
collecting evidence of impact? How to ensure that the subjects supply 
valid and reliable data? How to guard against mutual abuse of the 
evaluator by the subjects, and of the subjects by the evaluator 
perhaps a more frequent occurrence? 

(vi) What were the assumptions made and procedures followed in develop- 
ing a generalized set of evaluation possibilities within the overall 
program system of literacy? 

(vii) Who are the stakeholders with whom the evaluation plans must be 
negotiated to develop situation-specific evaluation agendas? 

(viii) Was there an understanding of what is involved in the utilization 
of evaluation results in regard to drawing recommendations from evalua- 
tion findings and to linking information back to decision making at 
clearly specified decision points? 

(ix) Are the data collected reliable and valid? Are these stored for 
iterative retrieval for use whenever needed? 

(x) Has the program system developed into a system which feels account- 
able to the various stakeholders and constituencies concerned with the 
program? 
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The preceding list of questions represents the theoretically 
possible according to just one individual specialist, (Hopefully a 

r 

good one!) Another literacy expert could come up with a somewhat 
different list, though overlap between any two such lists will be 
impossible to avoid. 

What is theoretically possible is not what evaluation specialists 
will want to do in full* They will have to make short lists that seem 
to meet tl^'r priority needs, within the real life and time of their 
programs? And that is not all. The lists drawn by technical profes- 
sionals may (if not always will) have to be approved, adapted and some- 
times vetoed by political actors involved in programs or concerned 
about their consequences, 

Evaluators as technicians have to be convinced that the task of 
developing these lists is not useless simply because political actors 
will be involved and will not gratefully accept all their suggestions 
so carefully crafted. Evaluation planning is not all logic, but it 
need not be all politics either. By developing good evaluation agendas 
that meet the criteria of need, practical feasibility and econorny, 
evaluators can make logic a part of the politics of evaluation; they 
can assist political actors in making sensible decisions and make it 
politically expensive for them to be merely political. 

It is also good to be reminded that individual evaluation studies 
should also be negotiated with the beneficiaries of programs. They 
should have a say in what should be evaluated* They should know why 
something is important to evaluate and how they can help in holding a 
mirror to all those engaged in the process of making social change. 



SECTION III 



FROM GENERAL EVALUATION POSSIBILITIES TO 
SITUATION-SPECIFIC EVALUATION AGENDAS 
AND APPROACHES TO EVALUATION 

In Section II, we have presented a way of delineating the total 
program system of literacy (or another nonformal education program) 
in design terms — that is, using system thinking and the four para- 
meters of the systems theory: contexts, inputs, processes and outputs. 
Such descriptions were then used to generate a whole series of possible 
evaluation questions which could possibly be raised by program special- 
ists and other decision makers. 

In this brief section, we like to make the point that evaluators 
can not be satisfied with such long and general lists and will have to 
go from the general to the specific. From a general set of possible 
evaluation questions, they will now have to develop specific evaluation 
agendas that make sense in the context of their particular program and 
its information needs, in a particular poMtical and cultural context, 
at a particular historical time. 

The criteria used in the selection of evaluation questions will 
have to be functional (the evaluation must generate the information 
and evidence needed by decision makers to perform their functions 
better), but evaluators will also have to be politically sensitive, 
and evaluation questionr> asked will have to be feasible in relation to 
available resources and infrastructures. 

Questions selected for evaluation could be fitted into a matrix 
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such as the one shown on the following page to see if all aspects of 
the program have been covered. (See Table III.l) We are not suggesting 
that aV[ the cells in the matrix should be filled by some evaluation 
question or another. We are only saying that this matrix can be used 
to check an evaluation agenda for adequacy and comprehensiveness. 

Once an evaluation agenda is available, this must go through the 
process of negotiation, on the one hand, with all the stakeholders 
within the policy and planning, system and, on the other hand, with the 
beneficiaries of the program or project. What does it mean to negotiate 
with policy makers? What does it mean to negotiate with the bene- 
ficiaries of the program? These questions will be discussed briefly in 
the section on "Evaluation Management." 

Towards an Operational ization of Evaluation 

For an operationalization of the evaluation plans, that is, to 
implement each evaluation study included in the agenda, each evalua- 
tion question should be analyzed with the help of a chart such as one 
produced below as Table III. 2. This exercise will force discussion on 
many imoortant issues that must be settled for the formal development 
of an evaluation proposal which has been discussed more fully else- 
where. (See H.S. Bhola, Evaluating Development Training Programs , 1982. 
Chapter VI.) 



Table IIIJ: A Matrix to Check for Adequacy and Comprehensiveness of the Evaluation Agenda 





Policy/ 

Planning 

Subsystem 


Administrative/ 
Mobil izational 
Subsystem 


Technical 
Resources and 
Instructional 
Development 
Subsystem 


Teaching/ 

Learning 

Subsystem 


Post/ 

Literacy 

Subsystem 


Evaluation 
Subsystem 


CONTEXTS 








1,2,3 






INPUTS 








1,2,3,4 


1,2 




PROCESSES 


1 




1 


1,2 







OUTPUTS 1 
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e III. 2: A Tool for the Operational ization of the Evaluation Effort 



Evaluation Evaluation Evaluation 
Question, x Question, y Question, z 

Evaluation Question/ 
Aspect/Objective 



Concept Analysis and 
^ Indicators 

Llnit(s) of Analysis and 
Sources of Data 



Instrumentation and Data 
Collection Approaches 



Remarks on Related 
Practical Considerations 



Models and Methods of Evaluation 
in Literacy Programs 

In the choice of models and methods of evaluation of literacy 
programs, one must learn to be eclectic. One must use what we have 
elsewhere described as the 3-S (Situation-Specific Strategy) model. 
The evaluator must begin with the problem and then use or adapt the 
model or models that will help in the systematic handling of the 
problem and its solution. 

In a siiiiiidr vein, one must choose s ituation-specific methodologies 
for the collection of data. In some cases, classical and quantitative 
approaches to data collection may be appropriate. In another setting, 
the information may be best collected through use of naturalistic and 
qualitative methodologies. In yet another case, evaluative information 
may be available through the analysis of the discourse of those involved 
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in the program as teachers or as learners. An analysis of the discourse 
of the peoples (without the use of questionnaires and survey instru- 
ments) can teach us a lot about processes and impact of our programs. 

One must also remember that all evaluation will not be data based. 
That is, some part of evaluation (and an importc.nt part of it) may not 
be empirical but analytical . It may be table-top analysis involving 
historical methodology, content* analysis and examination of policy 
statements and documents, 



Evaluation of Impact: 
The Case of Tanzania 

We are all aware of the work done in Tanzania in the area of 
literacy promotion during the 1970s. Recently,^ the department of 
adult education in Tanzania circulated a paper which -is self-- 
explanatory: 

CONCEPTUALIZATION OF EVALUATION OF THE 
IMPACT OF FUNCTIONAL LP/ERACY IN TANZANIA 

As you already know, in the past twelve years Tanzania has been 
engaged in a nation-wide literacy campaign. This campaign which has 
been conducted on the concept of functional literacy is an effort towards 
combating illiteracy anfong adults who form an essential portion of popu- 
lation in overall political, social and economic development. 

This struggle started on experimental basis with the then UNDP/ 
UNESCO Work-Oriented Adult Literacy Pilot Project in pursuit of the 
resolution passed by the Tehran Conference in 1965. When the World was 
celebrating 1970 as the Education Year, His Excellency The President 
Nkalin)u uulius K. Nyerere declared the year as the Adult Education Year 
in Tanzania. Following the encouraging results, ever since, there have 
been continuous efforts to eradicate illiteracy. 

Since that time, in Tanzania the literacy campaign has been 
regarded as an essential element in overall development and as such has 
been integrated in national development plans. The teaching of three 



The letter was dated May 26, 1982. 
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R*s went simultaneously with a curriculum package which included voca- 
tional training in agriculture, craftsmanship, health education, home 
economics and political education. Curriculum goals were geared towards 
higher production of food and cash crops and developing the rural popula- 
tion towards self-reliance. Another goal was to make the people 
politically conscious and understand their responsibility and role as 
members of a socialist nation with a policy of socialism and self- 
reliance. 

Since the inception of this programme, three National Literacy 
Tests have been administered in the years 1975, 1977, and 1981. These 
tests were executed with an intention of evaluating the extent to which 
illiteracy rate has been reduced. The 1975 literacy test revealed that 
the illiteracy rate had been reduced from 69% to 39%. In 1977 the 
percentage was reduced to 27Z and a further reduction down to 21% was 
realised in 1981 . 

Although the illiteracy rate now stands at 21%, there remains a 
need of evaluating the impact of the campaign on the community with 
special emphasis on adults who participated in the various programnes 
of the campaign. Currently, there are adults who are attending post- 
literacy programmes. Their abilities in reading, writing and arithmet.. 
have been assessed- but no attempt has been made to assess how the various 
theovetical and vocational skills imparted through the functional 
li Piracy classes, radio programmes, rural libraries, rural newspapers, 
discussion groups, vocational training centres, etc. have affected the 
attitudes, practices, behaviour, life outlook, political, social and 
economic performance of functional literacy graduates and the nation 
as a whole. 

Over the years we have been carrying the struggle with great 
enthusiasm. We may havs been successful in some cases and may have had 
failures in others. Numerous countries and organizations and various 
people have exchanged experience with us. Their k^^.jwledge and wisdom 
have contributed tremendously to our endeavour. We are now desirous of 
conducting an evaluation of the impact of functional literacy on our 
people. We intend to carry out a scientific investigation of real 
changes which have happened. We fundamentally feel that this exercise 
will determine the rate and extent of success or failure we have 
achieved. The findings of this evaluation will equally help us deter- 
mine guidelines for future planning. 

This evaluation of impact when completed will be a case of summative 
evaluation (evaluation that comes at the end of a program or an important 
phase of a program and sums up the whole experience). But these very 
questions must also be tackled as part of in-built formative evaluation 
(evaluation that can be used in the formulation and continuous improve- 
ment of the program). 
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A General Response 

Readers may be interested in the general response provided in 
regard to the evaluation plans described above: 

1- The overall major "evaluation study" should be seen as a collection 
of many interrelated tributary "evaluation studies." The tributary 
evaluation studies must, in each case, be carefully planned. A variety 
of data should be analyzed, using a variety of methods. One should 
envisage statistical analyses of correlations and trends, sample surveys, 
tracer studies of individual graduates of literacy classes, ethnographic 
studies of families and communities, and mass observations such as those 
used by Professor Madge in U.K. many years ago. 

2. The major Evaluation Study should accommodate many units of 
analysis: individual adults, families, groups, communities, institu- 
tions and the nation. The study of impact should not be done only in 
terms of the individual. 

3. The samples should be so chosen that comparison can be made 
between and among: 

Males and females 

Levels of literacy skills 

Rural and urban settings 

Occupational groups--cotton farmers, cattle growers, textile 

workers, etc. 

4- The sample of individuals chosen should additionally be time- 
sensitive. The testing years of 1975, 1977 and 1981 are obvious mile- 
stones. If at all possible, a distinction must, however, be made 
between the time when an individual became literate and the time when 
he or she was declared literate . 

5. In the study of impact, do not stop with economic impact only. 

Go 



i 



61 

Data should be collected and analyzed, among others, on locus of control, 
satisfaction v/ith life, participation in local institutions, health and 
nutrition, family planning attitudes, new motivations and aspirations for 
self and children, change or persistence of taboos and ethnic prejudices, 
understanding of political processes and of self-reliance, educational 
and cultural consumption, etc. 

6. I strongly recorrmend that statistical analyses be complemented ^by 
case studies using naturalistic anthropological approaches. Related 
with it is my suggestion that the Department should provide the leader- 
ship but not try to do all evaluation work by itself and under its own 
roof. All the different social scientific institutions in the country 
should undertake tributary evaluation studies as part of the overall 
education. 

There may be some ideas here that we can use as well in our own 
national settings. We are interested in impact at the various inter- 
mediate stages of our programs. 
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SECTION IV 



MANAGEMENT INFORMATION SYSTEM (MIS) 

In describing the needs for information in regard to the context 
of programs, their formation and implementation and feedback on their 
performance, we have been using the term eva'iuation in a generic sense. 
It should be clear, of course, that some of the answers to the questions 
listed above may be better answered by research than by evaluation (see 
the distinctions made in the earlier part of the paper). Also, some 
of the evaluations meiy be conducted as quick appraisals and through 
monitoring of the programs by actual visits or on the basis of periodical 
reports from the field. 

All of the processes of research, evalualiion, quick appraisals 
and monitoring can be helped by collecting and systematically organizing 
information routinely generated by and within the program. Such infor- 
mation can first be used in making everyday management decisions and 
then in evaluating effectiveness of decisions actions and their conse- 
quences. Such a system of information is called a management information 
system (MIS).^ 

The talk of an MIS raises in some minds the image of powerful 
computors, programmed to do impossible things, spilling out miles and 
miles of print-outs with tables, schematas and graphs, telling us what 

^A distinction must be made between "program management" and 
"evaluation management." While there is an overlap between problems 
in the two enterprises and some of the strategies of management are 
common to both systems, their frames of reference differ significantly. 

62 



63 



to do and thereby solving all our management problems. Nothing could 
be farther from the truth. * 

An MIS may have nothing to do with computors. It may be a manual 
system, that is, a set of tables and questionnaires filled at various 
points in time and files sequentially and systematically to provide us 
with serial data at varinus points in the life of a program. The MIS 
may indeed be of modest scope, collecting and storing data only on the 
most important variables impacting on the program. The data as stored 
may be in a disaggregated jrm and each time information is needed, 
data may have to be collated and fitted into a proper display. Whatever 
the size v^nd scope or nature of an MIS, it will not do our thinking for 
us. It will not tell us what to do. It will only describe the program 
in its various aspects. We will have to take it up from there and 
interpret this information and make management decisions for the 
maximization of results within our particular value frame. 

The Essential Nature of an MIS 

The essence of designing an MIS is to make the program to leave its 
footprints behind so that one can always find out how big the foot is, 
where the program has been, and where it is going. 

Too often in designing MIS's, literacy workers might make the 
error of collecting all possible numerical data but neglecting the 
collection of qualitative data as well as the "artifacts*' of tho program 
such as primers, graded books, teacher guides, teacher diaries, posters, 
photographs, etc. These kinds of data and evidence must not be . 
neglected. 
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The following considerations must enter the design of an MIS:^ 

(i) Choice of units and entities where change will appear . Such 
units may be individuals (learners, teachers, corflmunity leaders), groups 
(literacy classes, women*s clubs, discussion groups), institutions 
(health clinics, schools, rehabilitation centers), or communities or 
sub-cultures. Sometimes these may be physical entities, such as, 
primers, books, homes, fields, shops, wells, storage bins, etc. 

(ii) Choice of indicators of change . Indicators or signs that 
signify change in already selected units and entities must be chosen 
next. These indicators may be attendance in literacy classes, absence 
from the factory for reasons of ill -health, purchase of consumer goods, 
rise in productivity, etc. 

(iii) Economy in data collection . In designing an MIS one need 
not collect all possible data. Indeed only the minimum necessary 
information should become part of the MIS. 

(iv) Selection of points for origination, successive aggregation 
and storage of data . It should be clear where data will originate, at 
what successive levels and points data will be aggregated and where it 
will be finally stored for possible retrieval and use in decision making. 
This will require that data collection and data aggregation duties of 
various role performers in the program system are clearly identified; 
and the logistics of the physical transfer of data are managed 
effectively. 

^Excerpted from H.S. Bhola, Evaluating Development Training 
Programs . Bonn, FR6: Gernian Foundation for International Development, 
1982. (Pages 88-89). 
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(v) Design of proformas, tabulations, report forms, etc . Simplicity 
and standar^dization of instruments for data collection and reporting are 
important for the successful implementation of an MIS. 

Cvi) Timing and flow of data . Monitoring fails if it is not time- 
sensitive. Data should move in time series according to a determined 
pattern of flow — upwards, downwards and horizontally as required. 

(vii) Data audit . Data audits are important to ensure that there 
is no mis reporting of data and that the intent of various proformas in 
use in the MIS is we 11 -understood. 

(viii) Processing and fesdback . The data. collected at a 
particular level should be used to improve program decisions at that 
level before sending it upwards. When data is processed at the central 
level, or at one of the intermediate levels, feedback should be provided 
to all concerned. Only then can the total system grow in intelligence. 

The Case of a Unesco Project 

The Unesco Experimental World Literacy Program worked with a set 
of indicators that could usefully be examined by all those engaged in 
literacy evaluation. The Panel for the Evaluation of Experimental 
Literacy Projects, set up by Unesco in one of their meetings in Tehran. 
Iran, during September 1970, drew up a list, reproduced on the follow- 
ing pages, of eleven minimum and nineteen recommended indicators for 
use by the various projects. 



ERJ.C 



/ u 



9 * 



ERIC 



66 



LIST OF MINIMUM AND RECOMMENDED INDICATORS 

I BASIC STATISTICAL INDICATORS 

A. Quantitative aspects 

1. Minimum indicators 

1.1- Ratio of inscription in programme: Indicator is a ratio between 
the number of participants originally registered and number of 
places available. 

1.2. Rate of drop-outs 

1.3. Rate of attendance 

1.4- Time utilization: Indicator is a ratio between class sessions 
planned and actually held. 

2. Recommended indicators 

2.1. Rate of coverage: Indicator is a ratio between number of 
eligible clients and those actually covered in a programme. 

2.2. Rate of participation in final tests 

B. Qualitative aspects 

1. Minimum indicators 

1- 5. Literacy acquisition 

1.6- Acquisition of technical and professional knowledge 

2. Recommended indicator 

2.3. Acquisition of knowledge of socio-economic character 

C. Degree of adoption of writing, reading and calculating 

1. Minimum indicator 

1.7. Use of writing ability 

2. Recommended indicators 

2.4. Use of reading ability 

2.5- Use of arithmetical ability 

II INDICATORS PERTAINING TO ECONOMIC GROWTH AND DEVELOPMENT 

A- Production 

1. Minimum indicator 

T,z> l i^- Increase in production per capita 

2. Recommended indicators 

2.6. Quality of products 

2.7. Selling price 

. 2.8- Price (per quantitative unit) of the elements entering the costs 
of production 

B- Income — living standards 

1. Minimum indicator 

l.v. Increase in the number of durnble goods and improvements 
contributing to the standard of living 

2. Recommended indicator 

2.9. Increase of net global monetary income of individuals 

C- Production, servicing and transport equipment 
2. Recommended indicator 

2- 10- Increase in equipment for production, servicing or transport (in 

programmes for small agricultural producers) 

D. Socio-economic attitudes 
2. Recommerided indicator 

2-11- Importance of changes in the domain of socio-economic 
attitudes and in the individual's role in society. 

/ i 
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III ATTITUDES TOWARDS EDUCATION 

1. Minimum indicator 

1.10. Rate of scholarization of participants' children: This is a ratio 
between the total number of children of school age and those 
actually attending school. 

2. Recommended indicator 

2.12. Rate of interruption of schooling in participants' children: 
Ratio between the number of participants' children who inter- 
rupted their schooling and of interruptions in the total school- 
going population. 



IV PROFESSIONAL COMPETENCE 

2, Recommended indicators 

2.13. Adoption rate of recommended innovations 

2.14. Level of know-how in the fields covered by the programme 

2.15. Increase in desire for changes and technical innovations 



V RELATION TOWARDS MEANS OF MASS COMMUNICATION 

2. Recommended indicators 

2.16. Ownership of radios and television sets 

2.17. Preference for educational programmes 

VI HEALTH, HYGIENE AND SAFETY 

1. Minimum indicator 

1.11. Acquisition of icnowledge of health, hygiene and safeiy around 
home 

2. Recommended indicator 

2.18. Acquisition of knowledge of health, hygiene and safety in work 
settings 



VII COSTS AND COST-BENEFCT ANALYSIS 

2. Recommended indicator 

2.19. Per capita cost of functional literacy 



Some features of this list of indicators should be noted. First, 
it emerged from Unesco's concept of functional literacy and the objec- 
tives of functional literacy programs in both rural and urban settings, 
A more generalized model of literacy in development would have led to a 
somewhat different set of indicators. Secondly, the emphasis seems to 
be on indicators of inputs and outputs. Outputs are viewed in terms of 
information and attitudes learned and an attempt is made to measure them 
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in terms of appi ications of the information learned and the impacts of 
such applications on the life and work of participants. Thirdly, 
indicators have been divided into two clusters minimum and 
recommended. The implication is that, v/hile it might be advisable for 
a functional literacy project to use all of the thirty indicators 
suggested by the Unesco Panel, conditions will differ from program to 
program and from country to country, so that each project will have to 
develop its own evaluation policy. Fourthly, most of the indicators 
listed are not themselves directly observable and will have to be 
defined in terms of others that can be measured and weighed. 

Back in 1971, the Literacy Division of Unesco, Paris was interested 
in developing a Management Information System (MIS) to cover all the 
eleven or so projects all over the world. Indiana University was 
contracted to provide the outlines of a system that could be later 
tested-in-use and installed at the headquarters in ParisJ 

The skeleton of the Standardized Date Reporting System (SDRS) 
suggested by the Indiana University team to Unesco, Paris is reproduced 
below for it should suggest to readers some interesting ideas about the 
establishment of MIS's within literacy projects of our current concern. 

Professors H.S. Bhola and Michael Chiappetta of School of Education, 
Indiana University were in charge of the project at Indiana University. 
Late Dr. H.P. Saksena, tnen of Unesco Literacy Division, joined the team 
in Blooniington, Indiana for a short period of time. Faculties of 
Education, Psycholinguist'cs , Sociology and Economics at Indiana Univer- 
sity also provided guidance and help. 

The project involving the development of the Standardized Data 
Reporting System (SDRS) is described briefly in Michael Chiappetta, 
'*One Always Starts in the Middle: A Case Study Involving Unesco, 
Indiana University and Functional Literacy,'' Literacy Discussio n, Fall 
1974, pp. 351-372. 



ERIC 




\ 

Table IV J: Standardized Data Reporting on Functional Literacy Projects 
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Table IV. 2: Standardized Data Reporting on Functional Literacy 
Projects 
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I able IV. 3: Standardized Data Reporting on Functional Literacy 
Projects 
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Table IV. 4: Standardized Data Reporting on Functional Literacy 
Projects 



STAHDARDIZED DATA RZPORTI.SC 
OH FUNCTIONAL LITERACY PROJECTS 

TABLE lie 




Code 










(18) 


(19) 


(20) 


Soclo-cconoalc 
knovledst 


Httlth, Hygitne 
and Nutrition 


Any others 
(graphicecy, erClstlc deflpis) 






Cr 
Score 


Ltictl 
/ Vftlue 






Critical 
Score / Value 






Crltlcel 
Score / Veluc 


P 


r 


P 1 F 


P 


P 



























h'i ■ 

' t 



Table IV, 5: Standardized Data Reporting on Functional Literacy Projects 
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Table IV,6: Standardized Data ROiOrting on Functional Literacy Projects 
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Table IV,7: Standardized Data Reporting on Functional 
Literacy Projects 
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Table IV. 8: Standardized Data Reporting on Functional Literacy Projects 
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Some remarks are in order in regard to the design of forms and 
tables for an MIS and for a program in general. First, the forms and 
tables must not be so complicated that field workers can not fill those 
forms and tables. At the same time, all forms and tables, howsoever 
simple, must be explained to future users in some sort of training 
sessions. 

Second, the forms and tables used may collect both hard data and 
opinions, but the two must be clearly separated. One must not be 
allowed to be confused with the other. 

Third, there should be provision for collation of data at each of 
the various levels of toe program system. More importantly, this 
collation should be so organized that it compels understanding and 
use of data by program people handling the data. 

Some of the forms and tables, at the end, may indeed ask the 
worker direct questions such as these: Is there something new and 
unexpected in the data that you will want us to note? Have you decided 
to make any adjustments or changes in your work on the basis of this 
information? What are those adjustments or changes? 



SECTION V 

EVALUATION MANAGEMENT ^ \ 

Management is, In essence, the skillful use of available means. 
In modern management practice, management is a term with much more 
comprehensive import. It must involve, for example, not merely control 
of personnel butt leadership. The idea is that individual purposes of 
various role incumbents and organizational purposes can be reconciled 
and good management should achieve that reconciliation. Management as 
understood today also covers system design: invention of innovative 
roles and of administrative structures and mechanisms management 
by adhoceracy, for example — as well as creation and maintenance of 
proper organizational climate. Finally, management today must re-Jate 
to values larger than the narrow objectives of the organization and 
show social responsibility. 

There are some procedures that the manager can develop and 
install following some time-tested strategies and principles of manage- 
ment. But there are some overarching consideraticns in good management 
of evaluation systems. Two are most important: 

1 . Professional orientation of the system 

All those working within the evaluation subsystem and in the over- 
all program system should have a p rofessional orientation. What this 
means is that those involved in the program shoula look at social change 
as an experiment — an experim^^nt in humanity, an experiment in culture- 
making, but an experiment, nonetheless. This will mean, first, that 

1 78 



79 

thoughtful calculations are made of what means should be employed for 
what ends. Second, this will mean that results, both anticipated and 
unanticipated will be looked at and evaluated; and while such evaluation 
of results is undertaken, the emphasis will be to look for causes rather 
than culprits or heroes, 

2. An organizational climate of trust 

The able manager of evaluation subsystems must create a climate of 
trust within the evaluation subsystem and within the overall program 
system. Information is power. Once information is available, it can 
be both used and abused. Those who provide the information about 
themselves and their work must feel assured that it will not be used 
against them. There should be no incentives for people to falsify 
information. Those who receive information must trust the information 
they receive. They should not look at it as som^jthing made up, or 
something undependable , being produced for reasons other than functional. 
And in the process of the collection, collation, synthesis and movement 
of information, there should be no misuse designed to embarrass people, 
grouDS or parts of the organization. 

This professional orientation and climate of trust can not be 
ordered by memorandum. It will emerge over the months and years as the 
various organizational actors see the managers at work, listen to 
their words and watch their actions . 

Evaluation Subsystem: 

Organizational Design and Development 

Evaluation planning is not simply a matter of coming up with a 

list of evaluation questions. The evaluation planning process must 
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include the problems of management of the implementation process; and 
the planning later for the utilization of the evaluative information 
produced. 

The tasks in the management of evaluation include the development 
of an evaluation subsystem; linking this evaluation subsystem with the 
program system as well as with outside support institutions; recruitment 
and training of personnel to conduct the evaluation; resource generation 
and allocation; provision of logistical support, quality control and 
time budgeting of the enterprise; and ensuring appropriate and ethical 
use of information collected. 

(i) Developing an Evaluation Subsystem 

To make evaluation possible, an evaluation subsystem must be 
developed. This will typically involve the creation of a subculture of 
evaluation — many different people, playing many different roles at 
various levels of the program system. Some will be holding exclusively 
evaluation jobs, that is, they will not be engaged in program imple- 
mentation. Some will be program people, generating and collecting 
evaluation data and feeding it into the evaluation subsystem. In some 
cases, evaluation information will be collected and supplied by outside 
people, and institutions outside the boundaries of the literacy pro- 
gram. The essential point is that there will be a mix of full-time and 
part-time evaluation roles within all evaluation subsystems; and that 
there must be a clear understanding about the distribution of labor 
within these systems and subsystems. 

Readers' attention is invited to the section "The Right Conception 
of an Evaluation Subsystem" and Figure I.l including "The organization 
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of an overall program system showing the relationships between the 
evaluation subsystem and the program subsystem." 

(ii) Staffing and Recruitment 

* Staffing and recruitment of personnel for evaluation jobs 
should be undertaken carefully. Prospective candidates must be 
interviewed when the nature of the tasks involved in evaluation should 
be fully explained to them as well as the special obligations the 
job as evaluators. 

The tasks involved in the evaluation process are varied: logical 
tasks (involving the design of studies and instruments); social tasks 
(involving contacts and communications with individuals and comnrmi- 
ties); computing tasks (involving quantification of data, data colla- 
tion and, perhaps, some statistical analysis); and writing tasks 
(involving writing short and long reports based on field data). 

Evaluation also involves important obligations in regard to travel 
and work away from the desk in the office. Health of the evaluator and 
his family situation must permit travel away from home for long periods 
of time, on short notice. 

it is unlikely that in the developing areas of the world, trained 
evaluators will be easily available for hiring. There may be some 
candidates, sometimes, who have taken some "research courses" in a 
university. People with such backgrounds can be easily trained for 
their evaluation tasks. Soine caution is necessary, however. Some 
researchers have been socialized to do only research, dealing with 
questions they consider are significant. They think that whatever is 
not publishable is not worth doing. To them evaluation questions 
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dealing with an ongoing program are mere counting and describing. For 
them these tasks are not worth pursuing. We have to be careful that we 
do not recruit such people to join evaluation staff. 

(iii) * Training of Evaluation Staff 

The need for the training of evaluation staff is so obvious 
that it does not require any detailed comment. Task related training 
should be preferred to general training. Short training should be pre- 
ferred to long training. Training at home within the context of the 
program should be preferred to training abroad. When training abroad 
for long periods of time has to be provided to fit program needs, 
suitable contracts requiring trainees to serve the program on return 
must be developed, signed and made binding. 

(iv) Interfacing within the Program System and Training and 
Orientation of Program Staff 

More than once we have asserted that the evaluation subsystem 
is not made up only of the few people who carry the word evaluation in 
their design-:;cions. iJe have talked of the "right conception of an 
evaluation subsystem*' and suggested that while there will be some full- 
t^mp evaluation officers, all program officers will have to act as 
part-time evaluation officers. 

This should poinc up the "important need for developing organiza- 
tional interfaces and working relationships between the evaluation 
subsystem and the overall program system. This will mean that resources 
should be committed also to the training and orientation of program staff 
in the meaning, methods, and utilization of evaluation data. All 
participants in the system must understand clearly what their 
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obligations are, as also what are the rewards and credits due to them 
for their interest andlabor* Sharing the credit for evaluations 
completed is a very important consideration, for no one wants to put 
in hours and hours of labor so that others can write reports and advance 
their careers. 

(v) Linkage with Support Institutions 

Linkages and interfaces with support institutions are 
important as well. Evaluators of literacy programs, employed directly 
within projects and programs, may never be able to carry the whole 
burden of research and evaluation by themselves. They will need the 
help of universities to do the needed research and evaluation. They 
will need also the assistance of publishers, libraries and voluntary 
agencies. In the case of India, for example, it seems obvious that 
evaluation tasks will have to be divided between and, among the 
Directorate of Education; the departments of adult and nonformal 
education in the States; the infrastructure of State resource centers 
and some of the district resource centers; other departments dealing 
with development extension; and libraries, publishers and voluntary 
agencies. 

(vi) Decision Networks and Nodes 

Evaluation is too important to be left to evaluators alone! 
As part of the management of evaluation, mechanisms must be estab- 
lished that make decisions about what program decisions need to be 
serviced by evaluation; to Establish priorities among various needs; to 
establish evaluation teams and to give the go ahead on particular 
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evaluation studies. The various comnittees established for this purpose 
should have clear mandates. The various individuals who must convene 
those meetings and must implement decisions of those committees must 
also be identified and charged with their tasks clearly and unambiguously. 

(vii) Conimuni cation Within the System 

It is mcst important to ensure that no "rumor mill" develops 
around the evaluation tasks. Do not let people guess about what is 
going on; give them the information. Do not leave them to make their 
own uninformed and ideosyi.cratic interpretations based on partial data 
that they may happen to look at with someone, somewhere, at some time. 
Provide the total picture -- and as it develops. 

An in-house evaluation newsletter seems like an excellent idea. 
Such a newsletter should be more like a personal "letter" than like an 
impersonal "newspaper." The language snould be simple and the style 
personalized. And it must be informative. It should tell the reader^T 
all that is going on about evaluation -- objectives, methods, field 
problems, solutions developed to cope with them, and as far as possible 
early returns on data. 

(viii) Project Development and Team Building 

V Let us be reminded that as part of evaluation management two 
overlapping evaluation subsystems will have to be managed: 

(a) The continuous collection of data about the program as 
part of the MIS. (The concept .?nd design of MIS was discussed in an 
earlier section of this document.) This MIS will provide the steel 
frame for all evaluative activities such as monitoring, quick appraisals 
and special evaluation studies. 
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(b) The collection of data as part of specialized evaluation 
studies to meet specific information needs of program decisions. 

This section is addressed to (b) above. Once the evaluation 
question has been decided upon, those responsible for the management 
of evaluation should initiate the process of project development. An 
evaluation team should be named and brought together to plan, implement 
and report on the evaluation study. This team should spend a lot of 
time in conference — that is, in meetings where tasks are defined, 
assigned to different people and wherein tasks accomplished are 
reviewed by peers. 

It is for this team to also develop a PERT chart (a time schedule) 
for the accomplishment of variou?^ tasks that must be performed as part 
of the implementation of the evaluation project. 

(ix) Resources > Logistics, and Control 

Evaluation requires resources and these are not easy to 
obtain wnen they compete with program need^ within an overall condition 
of resource scarcity. Evaluation planners must learn to do most with 
the least of resources. This would mean that evaluators must make 
evaluation part of programing. Evaluation should be seen as routine 
as what one must do to implement a prograrr^ effectively. Such a stance 
will allow the evaluator to piggyback on the resources already avail- 
able to programs. 

In the Third World settings, where evaluation of literacy programs 
will often take place, logistics of sample selection, data collection 
and data analysis are going to present serious difficulties. Evalua- 
tors can not play God and will that there be telephones, and that there 
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be roads and that there be all the cars and landrovers needed for their 
trips upcountry. All they can do is play in a way that logistical 
problems are anticipated and as far as possible are accounted for. 

Finally, tnere is the problem of control: to ensure that the tasks 
get done; that the accomplishment is of high quality; that data are not 
fabricated or falsified; that tasks are completed on time; and that 
those participating in the process of evaluation can keep a high morale 

consider the work worth doing and can see its usefulness in the actual 
improvement of programs. 

The problem of control at the field level acquires a special 
aspect when we remember that many of the people working at the field 
level in the Third World will be vol-untary workers, working part-time 
and paid very small stipends. The extent of obligations one can put 
on such voluntary workers are quite a bit limited. 

(X) Monitoring as a Management Tool 

Monitoring described earlier as watching and checking upon 
an on-going program to detect flows and to provide corrective actions 
is an Important management tool. A good manager needs to know what is 
going on within the "living system" at any particular time. Monitoring 
must cover the total system of literacy and the various levels of 
action. The tabulation on next page (Table V.l) presents the scope of 
monitoring within a program system of literacy promotion. 

The entries in Table V.l are not complete and comprehensive. The 
table does indicate, however, the kinds of concerns, at the various 
levels of the program system, which must be covered in a monitoring 
effort. I 
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SECTION VI 

UTILIZATION OF EVALUATION RESULTS AND FINDINGS 

Evaluation lore is full of stories ^bout the nonuti 1 ization of 
evaluation studies; of evaluation studies that were rejected or ordered 
redone because tne powers that be did not like the results; or of 
evaluations that destroyed the careers and lives of e valuators who made 
the mistake of initiating and implementing them. 

Two different sets of problems seem to be involved. One set of 
problems can be subsumed under "Knowledge Production and Utilization." 
The other set of problems belong to the "Politics of Evaluation." 

Life is, of course, political. Evaluation (or for that matter 
program development ot research) can never be fully isolated from the 
political context. Again, neither individuals nor institutions are 
able to utilize in their work aJM the relevant knowledge that is indeed 
already available somewhere. But utilization oT evaluation findings 
can be considerably improved and its politics can be contained within 
limits by taking some simple steps. 

1 . Making Utilization of Evaluatio n 
Findings More Probable 

The process by which knowledge and information produced by 
researchers and* eval uators comes to be utilized by practitioners is 
by no means simple. It is indeed highly complex and we arc just now 
beginning to understand it. 

Some things are quite clear already. One important fact we have 

learned is that for knowledge to be utilized by the practitioner, it 
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must be appropriately processed . In other words, research must go 
through a "development'* phaf,e. In the area of evaluation, it would 
mean that the implications of the findings are worked out carefully 
for those who could use the evaluation findings and profit from doing 
so. 

A second important fact we have learned is that research utiliza- 
tion is not a rational process engaged in by reasonable men but that it 
is a social process , with a sociology of its own. The mere existence 
of evaluation findings is no guarantee that utilization will follow. 
There is nothing deterministic or even compelling about knowledge 
finding its use on its own. 

2. Limiting Some of the 
Politics of Evaluation 

It is well to understand that people working within organizations, 
as people generally, do not like surprises, nor do they enjoy being 
exposed to criticism. They certainly do not like to be attacked or 
"proved" to be less than efficient. And yet evaluation most often 
contains an element of surprise since it produces new information or 
throws new light on known facts. It does make some people open to 
criticism both justified and unjustified; and some people may indeed 
feel attacked since evaluation may show the consequences of the 
decisions they took or did not take, and may show the hollowness of 
some of the claims they might have made. This is all what makes 
evaluation political . 

When evaluation surprises, exposes, criticizes or attacks, the 
stakeholders may do one or more of the following: 

ERiC 9<i 



90 



(i) Hide or bury evaluation study informally through clever 
filemanship. 

(ii) Declare the study to be "classified" and contain the dis- 
cussion within the safe confines of the program bureaucracy. 

(iii) Call the evaluation study to be unauthorized and by impli- 
cation both illegal and mischevious. 

(iv) Attack the evaluators' intensions, program focus, choice of 
timing and also their methodology. 

(v) Attack the evaluators for not having provided the data earlier 
or to declare that the program conditions have changed since the 
evaluation study became available. 

(vi) Asking the evaluators to collect more data or to order a 
new study from a mre congenial group of evaluators. 

(vii) Reject the results of a study as being unusable within the 
prevailing value system of the society. 

As has been said earlier, evaluators can not play God and wish 
away the politics of evaluation. Nor can they guarantee that 
evaluation findings produced by them will indeed be put to work. 
However, some steps can be taken that will perhaps ensure that an 
evaluation effort does not become highly politicized and that chances 
of its utilization by decision makers are increased. 

( i ) Evaluation questions should be chosen in collaberation with 
future users . Evaluation (both internal evaluation and external evalu- 
ation) has to be decision-oriented. Evaluation issues and questions 
must be chosen according to the criteria of usefulness to the 
decision makers within the program system. Both the choice of questions 
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and the assignment of priorities among questions should be done with 
the decision makers themselves and within group settings. This will 
give "ownership" of the effort to the decision makers and "legitimiza- 
tion" to the evaluators. There is a slight risk in this collaboration 
in that that the decision makers may steer the evaluators away from the 
crucial questions. Evaluators will have to guard against this 
happening and will have to use the "objective fact" and the "neec}^ for 
information," already stated by the actors within the system in other 
contexts and at other times, as part of their argument. Such 
collaboration, happily, is bound to ameliorate both the political and 
the utilization problems, if not solve them completely. 

( i i ) A i'ormal ^prospectus (proposal) should be developed for the 
evaluation study . This needs to be done even in the case of quick 
appraisals. It must be ensured that the task of proposal writing 
does not become a substitute for the actual evaluation study. The 
proposal should be developed expeditiously and always on an emergency 
basis by a properly constituted committee that includes both 
evaluators and decision makers the future utilizers. A typical pro- 
posal should seldom take more than a 40-hour week. The first few 
hours of the proposal writing nieeting should make decisions about the 
statement of question, apfproatihes , samples, instruments and time 
schedule. Then tasks should be assigned appropriately to each member 
of the proposal writing comrT.ittee which should have between 4-6 members* 
After this initial meeting, ti>e group time should always be used for 
review and never for actual writing of text, or instruments. 

It is well to be reminded here that even in the so-called technical 
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area of evaluation methodology, there is no one correct way of doing 
things. If the program people for some reasons do not respect qualita- 
tive data, then offer to use both quantitative and qualitative 
approaches. Once the qualitative data has produced useful information, 
decision makers will be more sympathetic to this kind of data and to 
naturalistic inquiry methodology in general. If statistics demand only 
a small sample and the program people refuse to believe that such a 
small sample represents reality, then take a sample size that is 
acceptable. In other words, make samples both statistically and 
psychologically valid. 

Write a formal proposal. This will become a "contract" of sorts 
between and among all concerned. Give the names and designations of 
each and every one who collaborated in writing th^ proposal • Do not 
try to take all the credit for developing the proposal . Do not let 
people forget that this was a collaberative effort of the evaluators and 
the decision makers . 

(iii) Implement the study also as a collaborative effort , 
Evaluators must do what they are supposed to do and should not try to 
let others do all their work. However, they should make sure that the 
implementation of the study is seen as a collaborative effort. They 
absolutely must guard against monopolizing the evaluation effort to 
themselves as evaluators. 

The design of the instruments should be such that insights ejnerge 
for the program officials in the very process of data collection. 
Indeed, it me^y be possible to ask program officials at various Uvels to 
collate and interpret the data they collect and tnen send their 

ERIC 



<3t -o. 



93 

original disaggregated data up to the evaluators, along with their 
collations and interpretations. 

In the real world of evaluation and social action, evaluator may 
find that they have to make changes in their field work plans, sample 
sizes and sample sources, and even rewrite their evaluation questions 
and indicators. Such changes must always be discussed with the 
original proposal -writing committee. All concerned must understand 
that the changes made were imposed by the realities of the field. 
More importantly, it must be ensured that the integrity of the 
original evaluation intent and of the related study are not compromised. 

(iv) Complete the study on time and provide interim reports . 
It is absolutely important that an evaluation study is completed on 
time from the perspective of the decision maker. An evaluation 
study that is late is like an old newspaper. It is better to have a 
good enough study on time, than a perfect piece by job that was com- 
pleted too late. 

It is important, again, that interim reports are written and 
made available to decision makers. This should be done as many times 
as it makes sense. The nature of the evaluction question asked and 
the type of data collected will most likely impose a particular structure 
and phasing on the study. At the end of each natural phase, an interim 
report should be made available to decision makers. 

(v) Collective interpretation of results and development of norms . 
Evaluators must collate and analyze data and must prepare data displays, 
but they should conduct collective interpretations of data. Not only 
the program specialists but also the beneficiaries of programs should 
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be drawn into this collaborative interpretation. 

Even more important is the need to develop collectively a frame 
of reference for making sense out of the data in relation to the 
expectations for program performance . Is the average daily attendance 
of 15 in an adult class of 30 an acceptable condition? Is the increase 
in productivity of only 4% in contrast to the 15% expected a "satis- 
ficing" condition or a disaster? Is participant satisfaction with the 
program more important than the criteria of success laid down by an 
outside agency? 

(vi) Write multiple reports . Finally, there is the need to write 
not just one report but different reports for different constituencies, 
Eve'n more i:.portant, is the need to work out the program and administra- 
tive implications of an evaluation study and to translate the various 
implications into official orders and memoranda to which the program 
functionaries at various levels will respond. 

As indicated earlier, such steps will not make it absolutely 
certain that evaluation results are put to work by program specialists 
and other decision makers. It will, however, make it more probable. 



SECTION VII 



CONCLUDING REMARKS 

In the previous sections of this paper, we have discussed various 
related topics of interest to managers of programs and of evaluations. 

Evaluation planning was discussed in some detail in the first 
few sections. The point was made that an evaluation planner must first 
describe in design terms the program that is to be evaluated. Evalu- 
ation needs should be analyzed on the basis of such a description and 
then a situation-specific agenda should be developed to be negotiated 
with policy makers and political actors, on the one hand, and 
beneficiaries of the programs, on the other hand. 

The question can be asked: 

i 

If the 'politicians are going to decide, why should the technicians 
bother at all to make these evaluation plans and agendas? Why not 
let the powirs that be tell us what should be evaluated and what 
should be left alone? Such questions, not infrequent, are born of 
either impatience or cynicism, and arise in turn from a lack of under- 
standing of political processes and of the role of knoi^ledge in policy 
making. The professional evaluation planner, as technician, must dis- 
charge his or her duties in describing evaluation needs, stating 
significant evaluation questions and recommending and justifying 
priorities; and then must let the politicians discharge their duties. 
Only if the professional task has been done and done well will pro- 
fessional logic become part of the politics of decision making and make 
arbitrary decisions by politicians less likely and enlightened 
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choice of evaluations more and more likely. The professional evaluator 
in the process of negotiation will then not be professionally or 
personally compromised but will be able to make honest compromises 
without any loss of integrity. Through this paper we invite profes- 
sional specialists working on various literary projects, programs and 
campaigns to develop evaluation plans that are both technically suitable 
■ and politicaMy defensible and then put those plans through the processes 

of negotiation with the various stakeholders. 

In another section, we have dealt with the important question 
of developing management information systems (MIS's). We have 
emphasized the need to develop systems which collect the minimum 
necessary data and are so designed that they are most frequently put 
to work in decision making. 

The section on evaluation management may have repeated the 
obvious to many readers who themselves may have had long and significant 
experience of administration and management. Yet, many of these ideas 
need to be reinforced in a system that is always plagued by an "econofry 
of scarcities" and wherein the need for coordination and resource- 
sharing is an inevitable necessity. 

Finally, in the last section on utilization, we have included 
various practical suggestions to increase the probability that evaluation 
findings will be jsed and evaluators will not get hurt by the politics 
of evaluation. We have assumed a collaborative organizational style in 
making all these various suggestions. This assumption may not be true 
in many cultures of Asia, Africa and Latin America. Yet open and 
collaborative organizational styles will have to be learned both to 
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promote development and to be aole to evaluate our own efforts in 
honesty and with effectiveness* 
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